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VR at Valve 

• Began VR research 3+ years ago 

• Both hardware and software engineers 



• Custom optics designed for VR 

• Display technology - low persistence, global display 

• Tracking systems 

• Fiducial-based positional tracking 

• Desktop dot-based tracking and controllers 

• Laser-tracked headset and controllers 

• SteamVR API - Cross-platform, OpenVR 
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HTC Vive Developer Edition Specs 

• Refresh rate: 90 Hz (11.11 ms per frame) 

• Low persistence, global display 

• Framebuffer: 2160x1200 (1080x1200 per-eye) 

• Off-screen rendering ~1.4x in each dimension: 


VALVE 


• 1512x1680 per-eye = 2,540,160 shaded pixels per-eye (brute-force) 

• FOV is about 110 degrees 

• 360° room-scale tracking 

• Multiple tracked controllers and other input devices 
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VALVE 


© 
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Optics & Distortion (Pre-Warp) 


valve 



(Visualizing 1.4x render target scalar) 
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Optics & Distortion (Post-Warp) 


valve 


Warp pass uses 3 sets of UVs for RGB separately to account for spatial and chromatic distortion 



(Visualizing 1.4x render target scalar) 
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Shaded Visible Pixels per Second l VAL 

• 720p @ 30 Hz: 27 million pixels/sec 

• 1080p (5) 60 Hz: 124 million pixels/sec 

• 30" Monitor 2560x1600 @ 60 Hz: 245 million pixels/sec 

• 4k Monitor 4096x2160 @ 30 Hz: 265 million pixels/sec 

• VR 1512x1680x2 @ 90 Hz: 457 million pixels/sec 

• We can reduce this to 378 million pixels/sec (later in the talk) 

• Equivalent to 30" Monitor @ 100 Hz for a non-VR Tenderer 
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There Are No "Small" Effects 


valve 


• Tracking allows users to get up close to anything in the 
tracked volume 


• Can't implement a super expensive effect and claim "it's 
just this small little thing in the corner" 

• Even your floors need to be higher fidelity than we have 
traditionally authored 

• If it's in your tracked volume, it must be high fidelity 
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VR Rendering Goals 

• Lowest GPU min spec possible 

• We want VR to succeed, but we need customers 

• The lower the min spec, the more customers we have 

• Aliasing should not be noticeable to customers 

• Customers refer to aliasing as "sparkling" 


VALVE 


Algorithms should scale up to multi-GPU installations 

• Ask yourself, "Will 'X' scale efficiently to a 4-GPU machine?" 
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• VR at Valve 

• Methods for Stereo Rendering 

• Timing: Scheduling, Prediction, VSync, GPU Bubbles 

• Specular Aliasing & Anisotropic Lighting 

• Miscellaneous VR Rendering Topics 
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Stereo Rendering (Single-GPU) 

• Brute-force run your CPU code twice (BAD) 

• Use geometry shader to amplify geometry (BAD) 


VALVE 


• Resubmit command buffers (GOOD, our current solution) 


• Use instancing to double geo (BETTER. Half the API calls, improved 
cache coherency for VB/IB/texture reads) 

• "High Performance Stereo Rendering For VR", Timothy Wilson, San Diego 
Virtual Reality Meetup 
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Stereo Rendering (Multi-GPU) 


valve 


• AMD and NVIDIA both provide DX11 extensions to accelerate stereo 
rendering across multiple GPUs 

• We have already tested the AMD implementation and it nearly doubles our 
framerate - have yet to test the NVIDIA implementation but will soon 


• Great for developers 

• Everyone on your team can have a multi-GPU solution in their dev box 

• This allows you to break framerate without uncomfortable low-framerate VR 

• But lie to your team about framerate and report single-GPU fps :) 
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Prediction 


valve 


We aim to keep prediction times (render to photons) for the HMD and controller 
transforms as short as possible (accuracy is more important than total time) 

Low persistence global displays: panel is lit for only ~2 ms of the 11.11 ms frame 

22.22 ms of prediction 


11.11 ms 


VSync 


GPU 


Render 


\ 


Predict HMD Pose & 
Tracked Controllers 
22.22 ms 


H h 


11.11 ms 


VSync 


VSync 




Render Next Frame 


Image Sent To HMD Panels 


w 


/ 


Panels illuminated, 
user sees frame! 


NOTE: Image above is not optimal VR rendering, but helps describe prediction (See later slides) 
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Pipelined Architectures 

• Simulating next frame while rendering the current frame 


valve 


11.11 ms 


11.11 ms 


VSync VSync 

Game Simulation / Render Prep 


CPU 

CPU 


A 


Submit D3D Calls 


11.11 ms 


VSync 


VSync 


GPU 


Predict HMD Pose & 
Tracked Controllers 
33.33 ms 


\ 


Render 


Predict HMD Pose & 
Tracked Controllers 
22.22 ms 


Image Sent To HMD Panels 


W 


r 


Panels illuminated, 
user sees frame! 


We re-predict transforms and update our global cbuffer right before submit 
VR practically requires this due to prediction constraints 
You must conservatively cull on the CPU by about 5 degrees 
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Waiting for VSync 


valve 


• Simplest VR implementation, predict right after VSync 


• Pattern #1: Present(), clear back buffer, read a pixel 

• Pattern #2: PresentQ, clear back buffer, spin on a query 


• Great for initial implementation, but please DO NOT DO 
THIS. GPUs are not designed for this. 


• See John McDonald's talk: 

• "Avoiding Catastrophic Performance Loss: Detecting CPU-GPU 
Sync Points", John McDonald, NVIDIA, GDC 2014 
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GPU Bubbles 


VALVE 


• If you start submitting draw calls after VSync: 



• Ideally, your capture should look like this: 



(Images are screen captures of NVIDIA Nsight) 
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"Running Start" 


valve 


• If you start to submit D3D calls after VSync: 



• Instead, start submitting D3D calls 2 ms before VSync. (2 ms is a magic number 
based on the 1.5-2.0ms GPU bubbles we measured on current GPUs): 



• But, you end up predicting another 2 ms (24.22 ms total) 
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"Running Start" VSync 


valve 


• Question: How do you know how far you are from VSync? 


• Answer: It's tricky. Rendering APIs don't directly provide this. 

• The SteamVR/OpenVR API on Windows in a separate process spins 
on calls to IDXGIOutput::WaitForVBIank() and notes the time and 
increments a frame counter. The application can then call 
GetTimeSinceLastVSyncO that also returns a frame ID. 


GPU vendors, HMD devices, and rendering APIs should provide this 
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"Running Start" Details 


valve 




To deal with a bad frame, you need to partially synchronize with the GPU 

We inject a query after clearing the back buffer, submit our entire frame, spin on 
that query, then call PresentQ 

This ensures we are on the correct side of VSync for the current frame, and we can 
now spin until our running start time 


Predict 


Present 


Predict 


Present 


Present 


5— 

1 

Submit 

Spin 


Inject Event Query 


2 ms 


t 


h 


Wait until 2 ms before next VSync 


\ Clear , 

1 J n J ect Event Query 


Wait until 2 ms before next VSync 



GPU Rendering Stereo VR Frame #1 . GPU Rendering Stereo VR Frame #2 


GPU 


VSync 

I— 


11.11 ms 


Clear^ Event q uer y finishes 

VSync 

-HI- 


11.11 ms 


Cle 

VSync 

—I 
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Why the Query Is Critical 


valve 


• If a frame is late, the query will keep you on the right side of VSync 
for the following frame ensuring your prediction remains accurate 



VSync 


11.11 ms 


VSync 


11.11 ms 


Hh 


GPU Rendering Stereo VR Frame #2 

VSyr 


Clear^ Eyent Q uery pj n | S hes 

vs ' mc 11.11 ms 
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Running Start Summary 

• This is a solid 1.5-2.0ms GPU perf gain! 


VALVE 


• You want to see this in NVIDIA Nsight: 



• You want to see this in Microsoft's GPUView: 
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Aliasing Is Your Enemy l VAL 

• The camera (your head) never stops moving. Aliasing is 
amplified because of this. 

• While there are more pixels to render, each pixel fills a 
larger angle than anything we've done before. Here are 
some averages: 

• 2560x1600 30" monitor: ~50 pixels/degree (50 degree H fov) 

• 720p 30" monitor: ~25 pixels/degree (50 degree H fov) 

• VR: ~15.3 pixels/degree (110 degree fov w/ 1.4x) 

• We must increase the quality of our pixels 
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4xMSAA Minimum Quality 


valve 


• Forward Tenderers win for antialiasing because MSAA just works 


• We use 8xMSAA if perf allows 


• Image-space antialiasing algorithms must be compared side-by-side 
with 4xMSAA and 8xMSAA to know how your Tenderer will compare 
to others in the industry 

• Jittered SSAA is obviously the best using the HLSL 'sample' modifier, 
but only if you can spare the perf 
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Normal Maps Are Not Dead 

• Most normal maps work great in VR...mostly. 

• What doesn't work: 

• Feature detail larger than a few cm inside tracked volume is bad 

• Surface shape inside a tracked volume can't be in a normal map 

• What does work: 



Distant objects outside the tracked volume you can't inspect up close 
Surface "texture" and fine details: 
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Normal Map Mipping Error 


valve 



Expected 

glossiness 



Zoomed out 
super-sampled 
36 samples 


Incorrect 

glossiness 



Zoomed out 
normal map 
box filtered mips 


Blinn-Phong Specular 
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VALVE 


Normal Map Mipping Problems 

• Any mip filter that just generates an averaged normal loses 
important roughness information 


1 mini 


1 t\ t /tt / 


8x1 flat normal map 8x1 rough normal map 

These should NOT 
be the same! 

We've lost the 
roughness information! 

lxl flat normal map lxl rough normal map 


± 
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Normal Map Visualization 



4x4 Mip Visualization 


valve 



2x2 Mip 


4096x4096 Normal Map 
Fire Alarm 
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Normal Map Visualization 


valve 



16x16 Mip Visualization 8x8 Mip Visualization 
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Normal Map Visualization 


valve 



4096x4096 Normal Map 
Dota 2 Mirana Body 


4x4 Mip Visualization 
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Normal Map Visualization 


valve 



4096x4096 Normal Map 
Dota 2 Juggernaut Sword Handle 


4x4 Mip Visualization 
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Normal Map Visualization 


valve 



4096x4096 Normal Map 4x4 Mip Visualization 

Shoulder Armor 
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Normal Map Visualization 


valve 



4096x4096 Normal Map 
Metal Siding 


4x4 Mip Visualization 
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Roughness Encoded in Mips 

• We can store a single isotropic value (visualized 
as the radius of a circle) that is the standard 
deviation of all 2D tangent normals from the 
highest mip that contributed to this texel 

• We can also store a 2D anisotropic value 
(visualized as the dimensions of an ellipse) for the 
standard deviation in X and Y separately that can 
be used to compute tangent-space axis-aligned 
anisotropic lighting! 
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Final Mip Chain 

R = Roughness X G = Normal Y 


valve 


B = Roughness Y A = Normal X 
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Add Artist-Authored Roughness 


valve 


• We author 2D gloss = 1.0 - roughness 

• Mip with a simple box filter 

• Add/sum it with the normal map roughness at each mip level 

• Because we have anisotropic gloss maps anyway, storing the generated normal 
map roughness is FREE 



Isotropic Gloss Anisotropic Gloss 





Tangent V Roughness 
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Tangent-Space Axis-Aligned Anisotropic Lighting 


VALVE 



Standard isotropic lighting is 
represented along the diagonal 

Anisotropy is aligned with either of 
the tangent-space axes 


0.6 • | | 

0 9090011 

1.0 0.8 0.6 0.4 0.2 0.0 

Tangent U Roughness 


Requires only 2 additional values 
paired with a 2D tangent normal 
Fits into an RGBA texture (DXT5 
>95% of the time) 





GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


40 


Roughness to Exponent Conv ersion l VALVE 

• Diffuse lighting is Lambert raised to exponent 
(N.L k ) where k is in the range 0.6-1.4 

• Experimented with anisotropic diffuse 
lighting, but not worth the instructions 

• Specular exponent range is 1-16,384 and is a 
modified Blinn-Phong with anisotropy (more 
on this later) 

void RoughnessEllipseToScaleAndExp( float2 vRoughness, 

out float o_flDiffuseExponentOut , out float2 o_vSpecularExponentOut, out float2 o_vSpecularScaleOut ) 

{ 

o_flDiffuseExponentOut = ( ( 1.0 - ( vRoughness.x + vRoughness.y ) * 0.5 ) * 0.8 ) + 0.6; // Outputs 0.6-1.4 
o_vSpecularExponentOut .xy = exp2( pow( 1.0 - vRoughness .xy, 1.5 ) * 14.0 ); // Outputs 1-16384 

o_vSpecularScaleOut .xy = 1.0 - saturate( vRoughness .xy * 0.5 ); // This is a pseudo energy conserving scalar for the roughness exponent 



I I 

1.0 0.8 0.6 0.4 0.2 0.0 

Tangent U Roughness 
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How Anisotropy Is Computed 

Tangent U Lighting Tangent V Lighting 


( M° 


VALVE 


Final Lighting 
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Shader Code 

void RoughnessEllipseToScaleAndExp( float2 vRoughness, 

out float o_flDiffuseExponentOut, out float2 o_vSpeculanExponentOut , out float2 o_vSpeculanScaleOut 

{ 

o_flDiffuseExponentOut = ( ( 1.0 - ( vRoughness.x + vRoughness.y ) * 0.5 ) * 0.8 ) + 0.6; // Outputs 0.6-1.4 
o_vSpeculanExponentOut .xy = exp2( pow( 1.0 - vRoughness .xy, 1.5 ) * 14.0 ); // Outputs 1-16384 
o_vSpecularScaleOut .xy = 1.0 - satunate( vRoughness .xy * 0.5 ); // This is a pseudo energy conserving scalar for the roughness exponent 

} 

Isotropic Diffuse Lighting: 

float flDiffuseTerm = pow( flNDotL, fIDiffuseExponent ) * ( ( fIDiffuseExponent + 1.0 ) * 0.5 ); 

Anisotropic Specular Lighting: 

float3 vHa If AngleDi rials = normalize( vPositionToLightDirlals .xyz + vPositionToCameraDirWs .xyz ); 

float3 vSpecularNormalX = vHalfAngleDirWs .xyz - ( vTangentUWs .xyz * dot( vHa If AngleDi rials .xyz, vTangentUlals .xyz ) ); 
float3 vSpecularNormalY = vHalfAngleDirlals .xyz - ( vTangentVIals .xyz * dot( vHalfAngleDirlals .xyz, vTangentVIals .xyz ) ); 

float fINDotHX = max( 0.0, dot( vSpecularNormalX.xyz, vHalfAngleDirlals .xyz ) ); 
float flNDotHkX = pow( fINDotHX, vSpecularExponent .x * 0.5 ); 
flNDotHkX *= vSpecularScale.x; 

float fINDotHY = max( 0.0, dot( vSpecularNormalY.xyz, vHalfAngleDirlals .xyz ) ); 
float fINDotHkY = pow( fINDotHY, vSpecularExponent .y * 0.5 ); 
fINDotHkY *= vSpecularScale.y; 

float flSpecularTerm = flNDotHkX * fINDotHkY; 

Isotropic Specular Lighting: 

float flNDotH = saturate( dot( vNormallals .xyz, vHalfAngleDirlals .xyz ) ); 

float flNDotHk = pow( flNDotH., dot( vSpecularExponent .xy, float2( 0.5, 0.5 ) ) ); 

flNDotHk *= dot( vSpecularScale.xy, float2( 0.33333, 0.33333 ) ); // 0.33333 is to match the spec intensity of the aniso algorithm above 
float flSpecularTerm = flNDotHk; 


VALVE 
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Geometric Specular Aliasing 


valve 


• Dense meshes without normal maps also alias, and roughness mips can't help you! 


• We use partial derivatives of interpolated vertex normals to generate a geometric 
roughness term that approximates curvature. Here is the hacky math: 

float3 vNormalWsDdx = ddx( vGeometricNormalWs.xyz ); 
float3 vNormalWsDdy = ddy( vGeometricNormalWs.xyz ); 

float fIGeometricRoughnessFactor = pow( saturate( max( dot( vNormalWsDdx. xyz, vNormalWsDdx.xyz ), dot( vNormalWsDdy.xyz, vNormalWsDdy.xyz ))), 0.333 ); 
vRoughness.xy = max( vRoughness.xy, fIGeometricRoughnessFactor.xx ); // Ensure we don't double-count roughness if normal map encodes geometric roughness 



Visualization of fIGeometricRoughnessFactor 
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Geometric Specular Aliasing Part 2 

• MSAA center vs centroid interpolation: It's not perfect 


VALVE 


• Normal interpolation can cause specular sparkling at silhouettes due 
to over-interpolated vertex normals 


• Here's a trick we are using: 

• Interpolate normal twice: once with centroid, once without 

float3 vNormalWs : TEXCOORD0; 

centroid float3 vCentroidNormalWs : TEXCOORD1; 

• In the pixel shader, choose the centroid normal if normal length squared is 
greater than 1.01 

if ( dot( i.vNormalWs. xyZj i.vNormalWs.xyz ) >= 1.01 ) 

{ 

i.vNormalWs.xyz = i.vCentroidNormalWs. xyz; 

} 
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Outline 

• VR at Valve 


VALVE 


• Methods for Stereo Rendering 

• Timing: Scheduling, Prediction, VSync, GPU Bubbles 

• Specular Aliasing & Anisotropic Lighting 


Miscellaneous VR Rendering Topics 
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Normal Map Encoding 

• Projecting tangent normals onto Z plane only uses 78.5% of 
the range of a 2D texel 

• Hemi-octahedron encoding uses the full range of a 2D texel 

• "A Survey of Efficient Representations for Independent Unit Vectors", 
Cigolle et al v Journal of Computer Graphics Techniques Vol. 3, No. 2, 
2014 


valvTT 




(Image modified from above paper) 
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Scale Render Target Resolution 


valve 


• Turns out, 1.4x is just a recommendation for the HTC Vive (Each 
HMD design has a different recommended scalar based on optics 
and panels) 


• On slower GPUs, scale the recommended render target scalar down 

• On faster GPUs, scale the recommended render target scalar up 

• If you've got GPU cycles to burn, BURN THEM 
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Anisotropic Texture Filtering HAi 

• Increases the perceived resolution of the panels (don't 
forget, we only have fewer pixels per degree) 

• Force this on for color and normal maps 
• We use 8x by default 

• Disable for everything else. Trilinear only, but measure 
perf. Anisotropic filtering may be "free" if you are 
bottlenecked elsewhere. 
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Noise Is Your Friend 


valve 


• Gradients are horrible in VR. Banding is more obvious than LCD TVs. 

• We add noise on the way into the framebuffer when we have 
floating-point precision in the pixel shader 


float3 ScreenSpaceDither( float2 vScreenPos ) 

{ 

// lestyn's RGB dither (7 asm instructions) from Portal 2 X360, slightly modified for VR 
float3 vDither = dot( float2( 171.0, 231.0 ), vScreenPos.xy + g_flTime ).xxx; 
vDither.rgb = frac( vDither.rgb / float3( 103.0, 71.0, 97.0 )) - float3( 0.5, 0.5, 0.5 ); 
return ( vDither.rgb / 255.0 ) * 0.375; 

} 
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Environment Maps 


valve 


• Standard implementation at infinity = only works for sky 

• Need to use some type of distance remapping for environment maps 

• Sphere is cheap 

• Box is more expensive 

• Both are useful in different situations 

• Read this online article: 

• "Image-based Lighting approaches and parallax-corrected cubemaps", Sebastien 
Lagarde, 2012 
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Stencil Mesh (Hidden Area Mesh) l VALVE 

• Stencil out the pixels you can't actually see through the 
lenses. GPUs are fast at early stencil-rejection. 


• Alternatively you can render to the depth buffer at near z 
so everything early z-rejects instead 


Lenses produce radially symmetric distortion which means 
you effectively see a circular area projected on the panels 
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Stencil Mesh (Warped View) 


valve 
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Stencil Mesh (Ideal Warped View) 


valve 
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Stencil Mesh (Wasted Pixels) 


valve 
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Stencil Mesh (Unwarped View) 


valve 
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Stencil Mesh (Unwarped View) 


valve 
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Stencil Mesh (Final Unwarped View)[MEvi 
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Stencil Mesh (Final Warped View) 


valve 
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Stencil Mesh (Hidden Area Mesh) 



valve 


• SteamVR/OpenVR API will provide this mesh to you 

• Results in a 17% fill rate reduction! 

• No stencil mesh: VR 1512x1680x2 @ 90Hz: 457 million pixels/sec 


• 2,540,160 pixels per eye (5,080,320 pixels total) 


• With stencil mesh: VR 1512x1680x2 @ 90Hz: 378 million pixels/sec 
• About 2,100,000 pixels per eye (4,200,000 pixels total) 





GAME DEVELOPERS CONFERENCE® 2015 


MARCH 2-6, 2015 GDCONF.COM 


60 



Warp Mesh (Lens Distortion Mesh) l VALVE 
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Warp Mesh (Brute-Force) 
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Warp Mesh (Cull UV's Outside 0-1) 
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Warp Mesh (Cull Stencil Mesh) 
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Warp Mesh (ShrinkWrap) 
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15% of pixels culled from the warp mesh 
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Performance Queries Required! 

• You are always VSync'd 

• Disabling VSync to see framerate will make you dizzy 


VALVE 


• Need to use performance queries to report GPU workload 


• Simplest implementation is to measure first to last draw call 


• Ideally measure these things: 

• Idle time from Present)) to first draw call 

• First draw call to last draw call 

• Idle time from last draw call to Present)) 
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Summary 

• Stereo Rendering 

• Prediction 

• "Running Start" (Saves 1.5-2.0 ms/frame) 

• Anisotropic Lighting & Mipping Normal Maps 

• Geometric Specular Antialiasing 

• Stencil Mesh (Saves 17% pixels rendered) 

• Optimized Warp Mesh (Reduces cost by 15%) 

• Etc. 


VALVE 
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VALVE 


© 


Thank You! 

Alex Vlachos, Valve 

Alex@ValveSoftware.com 



